Preserve nanosecond resolution when encoding/decoding times #7827

kmuehlbauer · 2023-05-08T13:08:34Z

Closes nanosecond precision lost when reading time data #7817,
closes Numpy raises warning in xarray.coding.times.cast_to_int_if_safe #7942,
closes Fill values in time arrays (numpy.datetime64) are lost in zarr #7790,
closes [Bug]: reading NaT/NaN on M1 ARM chip #6191
closes Handle NaNs when decoding times (failures on riscv64) #7096
closes Differences on datetime values appears after writing reindexed variable on netCDF file #1064
Tests added
User visible changes (including notable bug fixes) are documented in whats-new.rst
New functions/methods are listed in api.rst

Closes #7098 as not needed, but worked around to preserve the fast int-based timedelta calculation.

kmuehlbauer · 2023-05-08T13:29:07Z

@spencerkclark I'd appreciate if you could have a look here. All but one test pass, but I can't immediately see what that test is doing. Looks like mismatched dtypes on the attributes. If you have any suggestions how to possibly improve, please let me know. I've not added tests here, yet.

kmuehlbauer · 2023-05-08T20:01:17Z

I've reset the order of coders to the initial behaviour. Instead the times are special cased in the CFMaskCoder. Locally it works, but I'll only trust the CI.

kmuehlbauer · 2023-05-08T20:22:28Z

All tests have passed. Rebased now on latest main. The issue described in #7817 is resolved. Ready for first reviews.

xarray/coding/variables.py

spencerkclark · 2023-05-08T21:35:16Z

Thanks @kmuehlbauer -- I just wanted to give you a heads up that I'm pretty busy this week. Hopefully I'll get a free moment to look at this more closely next week.

kmuehlbauer · 2023-05-09T03:51:39Z

Thanks for the heads-up, @spencerkclark. No worries, I need to apply some changes anyway as it turns out.

xarray/coding/variables.py

kmuehlbauer · 2023-05-10T20:27:08Z

@dcherian You were right from the beginning, changing order for decoding and handling _FillValue in CFDatetimeCoder seems to be one working solution with minimal code changes.

If the CI is happy I'll add tests to cover for the nanosecond issues in #7817.

xarray/coding/times.py

spencerkclark

Thanks @kmuehlbauer -- I'm catching up here a little bit. I think if we want to swap the order of decoding datetimes / timedeltas with decoding the mask information, we will also need to decode the _FillValue into something that has a datetime or timedelta type. We may not have great test coverage in this area.

As a concrete example I'm thinking about something like the following:

>>> import numpy as np; import xarray as xr
>>> times = [np.datetime64("2000-01-01"), np.datetime64("NaT")]
>>> da = xr.DataArray(times, dims=["time"], name="foo")
>>> da.encoding["dtype"] = np.float64
>>> da.encoding["_FillValue"] = 20.0
>>> da.to_dataset().to_netcdf("test-encode.nc")

On the current main branch we get the following behavior:

>>> xr.open_dataset("test-encode.nc").foo.values
array(['2000-01-01T00:00:00.000000000',                           'NaT'],
      dtype='datetime64[ns]')
>>> xr.open_dataset("test-encode.nc", decode_cf=False).foo.values
array([ 0., 20.])

With this branch we get:

>>> xr.open_dataset("test-encode.nc").foo.values
array(['2000-01-01T00:00:00.000000000',                           'NaT'],
      dtype='datetime64[ns]')
>>> xr.open_dataset("test-encode.nc", decode_cf=False).foo.values
array([ 0., nan])

Note the difference in the case where decode_cf=False. It seems like with this branch NaN is written out to disk when the fill value prescribes 20.0, which is incorrect. I think maybe the fact that NaN is being written out to disk is hiding the issue that the _FillValue needs to be decoded?

(There is probably a way to test this without doing I/O, but this was the quickest example that came to mind).

kmuehlbauer · 2023-05-19T12:57:31Z

Thanks @spencerkclark for taking the time. NaN has been written to disk (as you assumed). Let's have another try next week.

kmuehlbauer · 2023-05-22T09:24:47Z

@spencerkclark With current master I get the following RuntimeWarning running your code example:

on encoding (calling to_netcdf()):

/home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/times.py:618: RuntimeWarning: invalid value encountered in cast
  int_num = np.asarray(num, dtype=np.int64)

on decoding (calling open_dataset()):

/home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/times.py:254: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(
/home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/times.py:254: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(

The latter was discussed in #7098 (casting float64 to int64), the former was aimed to be resolved with this PR.

I'll try to create a test case using Variable and the respective encoding/decoding functions without involving IO (per your suggestion @spencerkclark).

kmuehlbauer · 2023-05-22T09:40:04Z

The example below is only based on Variable and the cf encode/decode variable functions.

import xarray as xr
import numpy as np

# create DataArray
times = [np.datetime64("2000-01-01", "ns"), np.datetime64("NaT")]
da = xr.DataArray(times, dims=["time"], name="foo")
da.encoding["dtype"] = np.float64
da.encoding["_FillValue"] = 20.0

# extract Variable
source_var = da.variable
print("---------- source_var ------------------")
print(source_var)
print(source_var.encoding)

# encode Variable
encoded_var = xr.conventions.encode_cf_variable(source_var)
print("\n---------- encoded_var ------------------")
print(encoded_var)

# decode Variable
decoded_var = xr.conventions.decode_cf_variable("foo", encoded_var)
print("\n---------- decoded_var ------------------")
print(decoded_var.load())

/home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/times.py:618: RuntimeWarning: invalid value encountered in cast
  int_num = np.asarray(num, dtype=np.int64)
/home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/times.py:254: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(
/home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/times.py:254: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(

---------- source_var ------------------
<xarray.Variable (time: 2)>
array(['2000-01-01T00:00:00.000000000',                           'NaT'],
      dtype='datetime64[ns]')
{'dtype': <class 'numpy.float64'>, '_FillValue': 20.0}
dtype num float64

---------- encoded_var ------------------
<xarray.Variable (time: 2)>
array([ 0., 20.])
Attributes:
    units:       days since 2000-01-01 00:00:00
    calendar:    proleptic_gregorian
    _FillValue:  20.0

---------- decoded_var ------------------
<xarray.Variable (time: 2)>
array(['2000-01-01T00:00:00.000000000',                           'NaT'],
      dtype='datetime64[ns]')
{'_FillValue': 20.0, 'units': 'days since 2000-01-01 00:00:00', 'calendar': 'proleptic_gregorian', 'dtype': dtype('float64')}

spencerkclark · 2023-05-22T11:58:18Z

Great, yeah, that's a nice example without writing to disk. Indeed I saw those warnings too, but omitted them in my earlier message to focus on the encoding issue (sorry about that). I agree that these are something we should address.

spencerkclark · 2023-05-29T13:00:30Z

One other tricky edge case that occurs to me is one where an extreme fill value (e.g. 1e30) is used for floating point fields. If we decode the times first, it might appear that the dates cannot be represented as nanosecond-precision values, but in reality they would be. We may need to think more about how to handle this edge case in addition to #7817.

spencerkclark

Beautiful @kmuehlbauer -- I appreciate those additional tests, the additional warning, and the further cleanup. One more suggestion for a minor tweak to a test, and I think this is ready for a what's new entry. Huge thanks for taking on this gnarly issue!

xarray/tests/test_coding_times.py

xarray/backends/netcdf3.py

kmuehlbauer · 2023-09-13T06:35:52Z

Thanks all for pushing this to a mergeable state. @spencerkclark 👍 💯

doc/whats-new.rst

Co-authored-by: Spencer Clark <spencerkclark@gmail.com>

dcherian · 2023-09-16T15:52:09Z

Amazing work, @kmuehlbauer and @spencerkclark . Thanks!

) * preserve nanosecond resolution when encoding/decoding times. * Apply suggestions from code review Co-authored-by: Spencer Clark <spencerkclark@gmail.com> * use emit_user_level_warning * move time alignment for nc3 to encode_nc3_variable * fix test for encode_cf_timedelta * fix CFMaskCoder for time-like (also allow timedelta64), add first tests * rename to _unpack_time_units_and_ref_date as suggested in review * refactor delta -> time_units as suggested in review * refactor out function _time_units_to_timedelta64, reorder flow and remove unneeded checks, apply filterwarnings, adapt tests * import _is_time_like from coding.variables * adapt tests, add _numpy_to_netcdf_timeunit-conversion function * adapt tests, add _numpy_to_netcdf_timeunit-conversion function * adapt test as per review, remove arm_xfail for backend test * add whats-new.rst entry * Update doc/whats-new.rst Co-authored-by: Spencer Clark <spencerkclark@gmail.com> * Update doc/whats-new.rst Co-authored-by: Spencer Clark <spencerkclark@gmail.com> * fix whats-new.rst --------- Co-authored-by: Spencer Clark <spencerkclark@gmail.com>

meteoDaniel · 2023-09-17T16:35:29Z

Awesome contribute @kmuehlbauer :)

github-actions bot added topic-CF conventions topic-cftime labels May 8, 2023

kmuehlbauer force-pushed the nanoseconds-resolution branch from 8aa6983 to 848ac09 Compare May 8, 2023 13:09

kmuehlbauer mentioned this pull request May 8, 2023

nanosecond precision lost when reading time data #7817

Closed

4 tasks

kmuehlbauer force-pushed the nanoseconds-resolution branch from 848ac09 to bb99536 Compare May 8, 2023 19:57

kmuehlbauer force-pushed the nanoseconds-resolution branch from 85c318c to b304aa0 Compare May 8, 2023 20:18

kmuehlbauer marked this pull request as ready for review May 8, 2023 20:22

Illviljan reviewed May 8, 2023

View reviewed changes

xarray/coding/variables.py Outdated Show resolved Hide resolved

kmuehlbauer force-pushed the nanoseconds-resolution branch from 5648c0f to 584b46e Compare May 9, 2023 14:01

dcherian reviewed May 10, 2023

View reviewed changes

xarray/coding/variables.py Outdated Show resolved Hide resolved

kmuehlbauer force-pushed the nanoseconds-resolution branch from 584b46e to 8750475 Compare May 10, 2023 20:17

kmuehlbauer mentioned this pull request May 11, 2023

Use numpy.can_cast instead of casting and checking #7834

Closed

3 tasks

kmuehlbauer commented May 11, 2023

View reviewed changes

xarray/coding/times.py Outdated Show resolved Hide resolved

spencerkclark reviewed May 18, 2023

View reviewed changes

tovogt mentioned this pull request Jun 29, 2023

Flood of cast warnings after improved hdf5 I/O CLIMADA-project/climada_petals#84

Open

preserve nanosecond resolution when encoding/decoding times.

db94bb9

kmuehlbauer force-pushed the nanoseconds-resolution branch from 94a7652 to db94bb9 Compare August 31, 2023 13:46

github-actions bot added topic-backends and removed topic-CF conventions labels Aug 31, 2023

Merge branch 'main' into nanoseconds-resolution

c1f810c

kmuehlbauer requested a review from spencerkclark September 12, 2023 14:04

This was referenced Sep 12, 2023

Handle NaNs when decoding times (failures on riscv64) #7096

Closed

Differences on datetime values appears after writing reindexed variable on netCDF file #1064

Closed

spencerkclark approved these changes Sep 13, 2023

View reviewed changes

xarray/tests/test_coding_times.py Outdated Show resolved Hide resolved

xarray/tests/test_coding_times.py Outdated Show resolved Hide resolved

xarray/backends/netcdf3.py Show resolved Hide resolved

kmuehlbauer added 3 commits September 13, 2023 07:51

Merge branch 'main' into nanoseconds-resolution

7f2b8b4

adapt test as per review, remove arm_xfail for backend test

d538ea9

add whats-new.rst entry

a75e4b8

kmuehlbauer added run-upstream Run upstream CI run-benchmark Run the ASV benchmark workflow labels Sep 13, 2023

spencerkclark reviewed Sep 14, 2023

View reviewed changes

doc/whats-new.rst Outdated Show resolved Hide resolved

doc/whats-new.rst Outdated Show resolved Hide resolved

kmuehlbauer and others added 4 commits September 14, 2023 05:38

Update doc/whats-new.rst

d4a71cd

Co-authored-by: Spencer Clark <spencerkclark@gmail.com>

Update doc/whats-new.rst

4dca66e

Co-authored-by: Spencer Clark <spencerkclark@gmail.com>

Merge remote-tracking branch 'origin/main' into nanoseconds-resolution

ee1b939

fix whats-new.rst

ebb00b8

kmuehlbauer added plan to merge Final call for comments and removed run-benchmark Run the ASV benchmark workflow labels Sep 16, 2023

Merge remote-tracking branch 'origin/main' into nanoseconds-resolution

c4d4a92

kmuehlbauer merged commit b6d4bf7 into pydata:main Sep 17, 2023
22 of 24 checks passed

kmuehlbauer deleted the nanoseconds-resolution branch September 17, 2023 08:15

kmuehlbauer mentioned this pull request Sep 17, 2023

Don't cast NaN to integer #7098

Closed

kmuehlbauer mentioned this pull request Sep 20, 2023

override units for datetime64/timedelta64 variables to preserve integer dtype #8201

Merged

4 tasks

spencerkclark mentioned this pull request Oct 4, 2023

time encoding fails for subdaily frequencies and days since #8271

Closed

4 tasks

spencerkclark mentioned this pull request Oct 22, 2023

Implement cftime vectorization as discussed in PR #8322 #8324

Open

tovogt mentioned this pull request Nov 1, 2023

suppress futile RuntimeWarnings from xarray CLIMADA-project/climada_python#742

Closed

13 tasks

mx-moth mentioned this pull request Jun 19, 2024

Can not save timedelta data arrays with small integer dtypes and _FillValue #9134

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preserve nanosecond resolution when encoding/decoding times #7827

Preserve nanosecond resolution when encoding/decoding times #7827

kmuehlbauer commented May 8, 2023 •

edited

Loading

kmuehlbauer commented May 8, 2023

kmuehlbauer commented May 8, 2023

kmuehlbauer commented May 8, 2023

spencerkclark commented May 8, 2023

kmuehlbauer commented May 9, 2023

kmuehlbauer commented May 10, 2023

spencerkclark left a comment

kmuehlbauer commented May 19, 2023

kmuehlbauer commented May 22, 2023

kmuehlbauer commented May 22, 2023

spencerkclark commented May 22, 2023

spencerkclark commented May 29, 2023

spencerkclark left a comment

kmuehlbauer commented Sep 13, 2023

dcherian commented Sep 16, 2023

meteoDaniel commented Sep 17, 2023

Preserve nanosecond resolution when encoding/decoding times #7827

Preserve nanosecond resolution when encoding/decoding times #7827

Conversation

kmuehlbauer commented May 8, 2023 • edited Loading

kmuehlbauer commented May 8, 2023

kmuehlbauer commented May 8, 2023

kmuehlbauer commented May 8, 2023

spencerkclark commented May 8, 2023

kmuehlbauer commented May 9, 2023

kmuehlbauer commented May 10, 2023

spencerkclark left a comment

Choose a reason for hiding this comment

kmuehlbauer commented May 19, 2023

kmuehlbauer commented May 22, 2023

kmuehlbauer commented May 22, 2023

spencerkclark commented May 22, 2023

spencerkclark commented May 29, 2023

spencerkclark left a comment

Choose a reason for hiding this comment

kmuehlbauer commented Sep 13, 2023

dcherian commented Sep 16, 2023

meteoDaniel commented Sep 17, 2023

kmuehlbauer commented May 8, 2023 •

edited

Loading